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ABSTRACT 


A  group  membership  protocol  provides  the  mechanisms  to  ensure  the  consistent 
group  views  among  a  group-oriented  distributed  processes.  The  protocol  is  required  to 
dynamically  re-configure  the  group  views  among  the  various  members  in  the  event  of  a 
change  to  the  group  due  to  a  new  member  joining  or  a  member  departing.  The  departure 
may  be  voluntary  or  involuntary.  The  protocol  must  provide  a  scheme  to  detect  the  failure 
of  any  of  the  members  and  re-configure  the  group.  Multiple  changes  to  the  group  must  be 
perceived  at  all  members  in  the  same  order. 

This  thesis  deals  with  a  particular  group  membership  protocol.  The  protocol 
structures  the  group  as  a  logical  ring.  Changes  to  the  group  are  accomplished  using  a 
two-phase  scheme.  The  agreement  phase  consists  of  circulation  of  an  agree  token. 
Processing  the  token  makes  a  pending  change  known  to  all  members.  The  commit  phase 
incorporates  the  changes  in  the  correct  order. 

This  thesis  presents  an  implementation  of  this  asynchronous  group  membership 
protocol.  The  main  feature  is  that  the  decentralized  nature  of  the  protocol  eliminates  the 
need  for  a  dedicated  coordinator  of  changes.  The  processing  requirements  for  the 
protocol  are  likewise  distributed.  The  processing  time  required  to  implement  a  change  to 
the  group  is  shown  to  have  a  linear  relationship  to  the  group  size. 
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I.  INTRODUCTION 


A.  BACKGROUND 

1.  Distributed  Computing 

Distributed  computing  is  at  the  forefront  of  today's  computing  research.  The 
increased  reliability  and  performance  has  resulted  in  distributed  computing  being  used  L 
various  applications  such  as  distributed  control  applications,  distributed  databases,  and 
real-time  settings  [1].  The  need  for  fault-tolerant  systems  is  particularly  important  to 
military  applications.  Such  applications  typically  require  fault  tolerant  algorithms, 
real-time  response,  on-line  reconfiguration,  and  other  schemes  to  increase  reliability. 
These  requirements,  however,  lead  to  significant  additional  complexity.  This  complexity 
arises  in  part  due  to  the  reliable  inter-processor  communication  required  to  implement  the 
distributed  processing.  Networks  are  inherently  unreliable  and  make  reliable  application 
level  message  passing  a  non-trivial  task.  One  of  the  primary  requirements  of  reliable 
distributed  applications  is  a  reliable  multicast  communication  primitive.  A  group 
membership  protocol  simplifies  the  construction  of  a  such  a  primitive  [2]. 

2.  Group  Membership 

Cooperating  processes  constituting  a  single  application  share  resources  and 
constitute  a  process  group.  Underlying  the  consistent  behavior  of  such  a  group  is  the 
requirement  that  all  members  of  the  group  have  implicit  knowledge  of  all  other  members 
in  the  group.  Additionally,  all  members  must  perceive  changes  to  the  group  in  the  same 
order.  Group  membership  will  change  as  processes  join  ir  depart  the  group.  Departures 
may  be  voluntary  or  involuntary  as  in  the  case  of  a  failure.  An  additional  requirement  is 
the  timely  detection  of  members  failing.  In  order  to  enhance  the  robustness  of  the  system, 
intra-member  monitoring  must  occur.  This  can  be  a  simple  exchange  of  messages 
indicating  that  the  process  being  monitored  is  still  "live." 
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Historically,  protocols  solving  the  group  membership  problem  have  been  of  a 
centralized  design.  One  member  acted  as  the  group  host  and  had  the  responsibility  of 
monitoring  all  subordinate  members.  All  changes  were  detected  by  the  host.  Upon  a 
change  to  the  group  view,  the  host  broadcast  the  new  group  view  to  all  members  of  the 
group.  Obviously,  problems  can  arise  if  the  host  itself  fails.  Voting  must  occur  in  order  to 
elect  a  new  host  with  the  added  overhead  associated  with  the  voting.  Additionally,  the 
processing  requirements  are  unequally  distributed  between  the  members  of  the  group  as 
the  host  takes  a  major  share  of  the  load. 

B.  SCOPE  AND  CONTRIBUTION 

In  this  thesis,  an  implementation  of  the  decentralized  asynchronous  membership 
protocol  is  presented.  The  protocol  was  originally  presented  in  [3]  and  [4].  It  was  further 
refined  and  partially  implemented  in  [5].  This  thesis  presents  a  brief  overview  to  the 
protocol.  A  more  detailed  explanation  of  the  protocol  and  definitions  can  be  found  in  [5]. 
This  thesis  covers  additional  refinements  to  the  protocol  necessary  to  correct  flaws 
discovered  in  the  coding  and  testing  phase  of  development.  Additionally,  the  performance 
of  the  protocol  on  an  Ethernet  Local  Area  Network  (LAN)  is  presented  and  discussed. 

C.  ORGANIZATION  OF  THE  THESIS 

The  thesis  is  divided  into  six  chapters.  Chapter  n  presents  an  overview  of  the  basic 

operation  of  the  protocol.  Chapter  HI  discusses  the  changes  required  to  implement 
improvements  to  the  protocol.  Chapter  IV  covers  possible  further  refinements  to  the 
protocol.  In  Chapter  V,  the  performance  of  the  current  protocol  is  analyzed.  Chapter  VI 
presents  the  analysis  and  possible  future  areas  of  research.  The  code  developed  is 
included  in  the  Appendix. 
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II.  GROUP  MEMBERSHIP  PROTOCOL 


In  this  chapter,  the  group  membership  protocol  (GMP)  is  described.  The  original 
protocol  was  presented  in  [3]  and  further  developed  in  [5].  This  chapter  presents  an 
overview  of  the  protocol. 

A.  GROUP  MEMBERSHIP  PROTOCOL  OVERVIEW 

1.  Assumptions 

The  following  assumptions  are  made  by  the  GMP  in  order  for  proper 
implementation.  A  fully-connected  network  of  reliable  First-In  /  First-Out  (FIFO) 
communication  channels  connecting  operational  members  is  assumed.  All  failures  arc 
assumed  to  be  crash  or  fail-stop[(i\.  This  implies  that  a  message  sent  will  be  delivered 
unless  the  recipient  has  failed.  However,  there  is  no  upper  bound  on  the  time  of  delivery. 

Multiple  changes  to  the  membership  are  allowed  simultaneously.  However,  the 
changes  are  committed  one  at  a  time  and  in  the  same  order  at  all  members. 

A  member  is  added  to  the  group  when  a  join  is  processed  and  is  removed  when 
a  failure  is  perceived. 

The  group  name  is  assumed  to  be  public.  Those  elements  that  may  wish  to 
become  members  by  joining  the  group  have  access  to  the  group  file  which  contains  the 
current  group  view.  A  prospective  member  searches  for  the  file  on  a  given  site.  The 
group  view  is  extracted  and  the  prospective  member  sends  the  joinreqst  to  the  appropriate 
address. 

The  protocol  maintains  three  main  databases  at  each  member;  the  membership 
list  (group  view),  status  table  and  token  pool.  Each  has  a  separate  database  manager  to 
ensure  mutual  exclusion  to  all  processes  needing  the  services  of  the  database. 

2.  Overview 

The  GMP  guarantees  that  the  group  view  changes  occur  in  the  same  relative 
sequence  at  all  operational  members  of  the  group. 
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The  most  significant  feature  of  the  GMP  is  the  decentralization.  No  single 
member  is  responsible  for  detecting  a  change  to  the  membership  nor  guaranteeing  group 
view  consistency  among  the  group  members.  A  logical  ring  is  used  to  implement  both  of 
these  functions  in  a  distributed  manner.  The  logical  ring  is  a  circular  ordering  of  the 
members  of  the  group. 

The  physical  location  of  the  member  has  no  relation  to  the  ordering  of  the 
logical  ring.  Within  the  ring  structure  the  direction  of  traversal  was  arbitrarily  chosen  as 
clockwise.  Given  the  structure  each  member  only  monitors  its  anti-clockwise  neighbor, 
the  acwnbr.  The  acwnbr  responds  to  a  status  query,  statusqry,  with  a  status  report, 
statusrpt.  Likewise,  it  sends  a  statusqry  to  its  acwnbr.  Thus,  every  member  monitors 
only  one  other  member  of  the  group  and  is  itself  monitored  by  it  clockwise  neighbor, 
cwnbr. 

Consider  a  five  member  group.  Process  po  was  the  initial  member  of  the  group. 
The  other  members  joined  in  an  order  such  that  p^  is  the  acwnbr  of  p,,  p,  is  the  acwnbr  of 
Pj,  and  so  on.  Member  Pj  sends  a  statusqry  to  Po,  which  responds  with  a  statusrpt. 
Likewise,  p^  sends  a  Jtatusqry  to  pp  This  is  illustrated  in  Figure  1.  For  clarity,  only  the 
monitoring  and  response  is  shown  for  the  first  set  of  neighbors. 

The  ring  configuration  changes  as  the  group  membership  changes.  The  ring 
starts  as  a  single  member  group  with  other  members  joining  in  some  arbitrary  order.  All 
changes  to  the  group  are  treated  in  a  similar  manner.  Members  wishing  to  join  an  existing 
group  read  the  group  membership  file  in  the  first  active  member  located  on  the  net.  The 
joining  member  then  sends  a  joirureqst  to  the  first  member  of  the  group.  If  the  initial 
parameters  message,  initparams,  is  not  received  in  a  reasonable  time,  the  request  is 
transmitted  to  the  next  member  in  the  group  view  file  until  all  members  have  been 
attempted  or  a  successful  join  has  been  completed. 

A  failure  is  considered  an  involuntary  depanure.  When  a  member  departs  the 
group  voluntarily,  it  simply  stops  responding  to  statusqrys.  It  will  then  be  perceived  as 
failed  and  subsequently  removed  from  the  group.  E>elayed  transmission  of  the  statusrpt. 
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failure  of  the  member  to  respond  to  the  statusqry  and  a  lost  statusrpt  will  all  result  in  the 
monitored  member  being  detected  as  failed 


Figure  1  A  Logical  Ring 


3.  Processing  of  Individual  Changes 

The  GMP  allows  for  a  two  phase  procedure  for  all  changes  to  the  group  view, 
the  agree  phase  and  the  commit  phase.  An  agree  token  is  circulated  around  the  ring. 
Once  the  originator  receives  the  token  via  the  ring,  all  members  have  processed  the  agree 
token.  At  this  point  the  agree  token  is  converted  to  a  commit  token,  and  the  change 
agreed  upon  is  committed  by  each  member  as  the  token  is  circulated  around  the  ring.  The 
protocol  ensures  that  each  token  is  received  by  all  members,  processed  only  once,  and 
never  lost  More  complete  descriptions  of  the  actions  required  by  the  different  phases  is 
covered  in  the  following  section. 


a.  Departure  Processing 

Once  a  member  perceives  the  departure  of  its  ccwnbr,  voluntary  or 
otherwise,  a  failagree  token  is  initiated.  The  failed  member  is  added  to  the  status  table 
with  a  failagree  status.  The  token  is  incorporated  into  the  token  pool  and  transmitted  via 
the  FIFO  channel  to  the  cwnbr.  Similar  processing  occurs  at  members  receiving  a 
failagree  token  for  the  first  time.  Once  the  token  has  been  received  by  the  originator,  the 
agree  phase  has  been  completed.  The  failed  member  is  removed  from  the  status  table  and 
group  view.  The  token  pool  is  purged.  The  failcomit  token  added  to  the  token  pool  and 
transmitted  around  the  ring.  When  a  member  receives  a  failcomit  token  for  the  first  time, 

similar  processing  occurs. 

b.  Join  Processing 

The  protocol  maintains  a  logical  marker  between  the  first  and  last  members 
to  join  the  group.  The  first  member  is  called  the  host.  A  new  member  will  always  join  the 
group  as  the  acwnbr  of  the  host.  The  host  has  the  responsibility  of  initiating  the  joinagree 
token  for  the  new  member.  However,  the  host  may  not  be  the  member  of  the  group  that 
receives  a  join  request  message  from  the  new  member.  In  this  case,  the  message  is 
converted  to  a  token,  forwarded,  and  the  new  member  is  added  to  the  status  table.  Once 
the  host  has  received  either  a  Joinreqst  token  or  message,  it  initiates  a  Joinagree  token. 
The  host  then  adds  the  token  to  the  token  pool,  the  member  to  the  status  table,  and 
transmits  the  token  to  its  cwnbr.  Similar  processing  occurs  when  a  member  receives  a 
joinagree  token  for  the  first  time.  When  the  host  receives  the  Joinagree  token  via  the 
token  ring,  it  initiates  the  commit  phase. 

The  Joincomit  phase  consists  of  purging  the  token  pool,  incorporating  the 
joincomit  token  in  the  token  pool,  adding  the  new  member  to  the  group  view,  and 
forwarding  the  token.  Additionally,  the  host  will  transmit  the  status  table,  group  view, 
and  token  pool  to  the  new  member  in  the  initial  parameters  message,  initparams.  This  is 
accomplished  by  IntegrateM ember. 
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III.  PROTOCOL  CHANGES 

This  chapter  describes  the  revisions  to  the  group  membership  protocol  proposed  in 
[5].  Modifications  discussed  include  lost  messages,  delayed  transmission,  lost  token 
acknowledgments,  and  proper  termination  of  the  agree  phase. 

A.  LOST  INITIAL  PARAMETERS  MESSAGE 

1.  Problem 

Consider  a  lost  initial  parameters  initparam  message.  After  the  initparam 
message  is  sent,  the  joining  member  is  regarded  as  part  of  the  group  by  the  sender  even  if 
it  is  lost.  However,  the  group  membership,  token  pool  and  status  table  are  not  accurately 
reflected  in  the  joining  member's  local  database.  There  was  no  mechanism  for 
re-transmission  of  the  initparam  message,  nor  was  it  possible  to  recreate  the  information 
locally. 

2.  Solution 

The  status  reporter  is  required  authenticate  the  group  membership.  Reports  are 
generated  in  response  to  status  queries  from  only  those  members  that  are  in  the  group  or 
are  joining  the  group.  Status  queries  from  outside  the  group  are  ignored.  See  Figure  2. 

ReportStatus  process  at  p. 

1  if  (not  blocked  by  IntegraUMember) 

2  if  (querying  member  e  GVp^  or  has  joinagree  status) 

3  p^^  =  querying  member 

4  send  status  to  p^, 

5  if  (previous  querying  member  =  p^,) 

6  send  TokenPoolip)  to  p^, 

7  end 

8  end 

9  end 

end  ReportStatus 

Figure  2  Reporting  of  Status 
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3.  Justification 

A  lost  initparam  message  will  result  in  the  new  member  failing  to  respond  to 
the  host's  first  statusqry.  Upon  time  out,  the  new  member  will  be  considered  a  failed 
process.  The  host's  original  acwnbr  will  then  be  monitored  anew  by  the  host.  The  new 
member,  never  having  received  the  initparam  message  will  time  out  on  the  join  request 
It  will  then  attempt  to  join  again. 

By  the  time  the  initparam  message  is  transmitted,  the  host's  original  acwnbr 
has  knowledge  that  the  new  member  is  now  joining  the  group.  Therefore,  with  this 
change,  the  host's  original  acwnbr  issues  status  reports  to  the  new  member's  queries. 

4.  Side  effects 

Such  group  authentication  will  prevent  multiple  switching  of  the  cwnbr.  If  the 
initparam  message  is  lost,  the  host’s  original  acwnbr  will  be  un-monitored  for  a  short 
time.  UjK  ^he  failure  detection  of  the  new  member,  the  host's  original  acwnbr  will  again 
be  monitored  by  the  host  process.  To  ensure  that  the  StatusReporter  responds  only  to 
members  within  the  current  group  view,  IntegrateMember  must  be  atomic  with  respect  to 
StatusReporter.  This  will  prevent  race  conditions  when  initparam  message  is  received, 
followed  by  an  almost  simultaneous  receipt  of  the  first  status  query  from  the  host  process. 

Figure  3  shows  the  process  dependencies,  while  Figure  4  depicts  the  specification  for 
IntegrateMember,  The  inter-process  dependencies  for  the  monitor  processes  are  shown 
in  Figure  5. 
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Figure  3  Integrate  Member  -  Process  Dependencies 


IntegrateMember 

1  if  (initial  parameters) 

2  send  blocking  message  to  status  reporter 

3  send  GVp,  to  group  view  manager 

4  send  unblocking  message  to  status  reporter 

5  send  STp^  to  status  table  manager 

6  send  TokenPoolip,)  to  token  pool  manager 

7  else 

8  get  GV„  from  group  view  manager 

9  get  S7pj  fixan  status  table  manger 

10  get  TokenPooKp)  from  token  pool  manager 

1 1  assemble  initparam  message 

12  send  initparam  message  to  new  member 

13  end 


Figure  4  Integrate  Member  Process  Specification 


Figure  S  Monitor  Process  -  Internal  Structure  and  Dependencies 


B.  DUPLICATE  PROCESSING 

1.  Problem 

The  protocol  was  designed  for  processing  in  a  member  to  be  concurrent 
Consider  AgreeProcessor  and  ComitProcessor.  It  is  possible  for  the  AgreeProcessor  to 
receive  an  agree  token  immediately  followed  by  an  external  token  pool  containing  the 
same  token.  The  token  may  be  converted  to  a  commit  token  and  forwarded  to 
ComitProcessor.  Due  to  context  switching,  processing  of  the  commit  token  may  not  be 
immediate.  Further,  AgreeProcessor  begins  processing  the  token  pool  which  contains  the 
cq)y  of  the  original  agree  token.  Since  the  processing  of  the  commit  token  has  not 
occurred,  it  is  possible  for  the  agree  token  from  the  token  pool  to  be  detected  as  requiring 
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conversion.  The  subsequent  processing  of  the  duplicate  commit  is  an  error.  Figure  6 
depicts  the  problem. 


Tims 


AgreeProcessor 

Tecdwe  failagree  token 
process  failagree 
send  init  failcomit 

receive  external  token  pool 
process  failagree 


ComitProcessor 


send  intt/oSeonur 


% 


error  condition 


receive  init  failcomit 

process  failcomit 

-  update  status 

-  purge  token  pool 

-  commit  change 


Figure  6  Error  Condition  Arising  from  Asynchronous  Programs 


2.  Solution 

In  order  to  solve  this  problem,  CommitProcessor  must  be  atomic  with  respect 
to  AgreeProcessor.  (see  Figures  7  and  8) 

3.  Justification 

Both  processors  use  the  same  databases.  Rejection  of  duplicate  tokens  depends 
upon  the  current  state  of  the  databases.  Since  token  rejection  is  accomplished  by 
AgreeProcessor,  and  ComitProcessor  updates  the  state  to  reflect  the  commit  in  progress, 
AgreeProcessor  must  not  begin  its  next  iteration  until  ComitProcessor  has  updated  the 
state.  ComitProcessor  does  not  fully  update  the  state  until  just  prior  to  transmitting  the 
commit  token  around  the  ring.  Refer  to  Figure  9,  lines  1-3. 
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Figure  7  Agreement  Processor  -  Process  Dependencies 
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Figure  8  Commit  Processor  -  Process  Dependencies 
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CommitChange  for  commitpj(p^)  at  p. 

/*  Depending  on  whether  a  join  or  departure  */ 

1  add  or  delete  p^  from  GVp. 

2  delete  p^  entry  from  STp, 

3  vn(p,)  vn(p.)  +  1 

4  delete  all  commit  tokens  received  before  agreepfp,)  from  TokenPoolip) 

5  if  (join  committed  &&  joinreqpfjp,)  e  TokenPool(p)  ) 

6  delete  joinreqp^p^) 

7  end 

8  add  commitpjip^)  to  TokenPoolip) 

9  delete  agreepjip^) 

10  if  (current  host  =  pj 

1 1  determine  new  p*„, 

12  end 

13  if  ((join  committed)  &&  (p^^,,  =  p,)) 

14  send  STp,,  TokenPoolip),  and  GVp^  to  acwnbrip) 

15  end 

16  send  commitpjip)  token  to  cwnbrip) 
end  CommitChange 


Figure  9  Actions  for  Committing  a  Change 

C.  INVALID  DELETE  TOKEN 

1.  Problem 

An  attempt  to  delete  a  nonexistent  joinreqst  token  from  the  token  pool  would 
result  in  an  error  condition  and  would  hang  the  process.  The  joinreqst  token  is  not  always 
present  in  the  token  pool.  The  token  occurs  only  if  a  joining  member  has  made  the 
request  to  a  member  that  is  not  the  host. 

2.  Solution 

Before  attempting  to  delete  the  Joinreqst  token,  the  token  pool  is  checked.  If 
the  token  is  present,  it  is  then  deleted.  Figure  9,  lines  5-7. 
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3.  Justification 

This  is  a  special  case  token,  and  does  not  always  occur  in  all  token  pools. 
Exception  handling  as  above  will  correct  any  incor.  sistencies  among  the  various  token 
pools. 

D.  LOST  TOKEN  ACKNOWLEDGMENT 

1.  Problem 

If  the  token  acknowledgment  is  not  received  by  the  sending  ^on/  process,  the 
token  will  remain  on  the  queue.  It  will  be  re-transmitted  every  time  that  front  receives  a 
message.  The  original  token  will  be  processed  at  the  receiving  end  and  further  receipts  of 
the  ‘'ame  token  will  be  detected  as  duplicates  and  discarded.  The  problem  lies  in  that  the 
queue  is  ..ever  cleared.  Therefore,  subsequent  tokens  will  be  blocked  behind  the  token  for 
which  the  acknowledgment  was  lost,  unless  a  failure  of  one  of  the  two  members  occurs. 

2.  Solution 

This  problem  is  solved  by  checking  the  serial  number  of  the  message  on  the 
receiving  end  of  the  FIFO  channel.  If  the  message  is  the  last  token  received,  the  token 
acknowledgment  is  re-transmitted . 

3.  Justification 

If  the  message  received  is  the  expected  token,  an  acknowledgment  is  sent.  The 
token  is  forwarded  to  the  appropriate  internal  sub-process.  If  the  last  token  received  is 
received  again,  a  token  acknowledgment  is  sent  back  and  the  duplic?te  token  is  discarded. 
This  mechanism  will  account  and  correct  for  lost  token  acknowledgments.  Figure  10, 
li.-.es  17-19. 
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FIFO  Channel  -  BACK  process 

1  Wait  for  a  channel  ready  to  ready 

2  if  (internal  channel  ready) 

3  if  (Status  Query) 

4  update  acwnbr 

5  send  StatusQuery 

6  else  if  (Initial  Parameters) 

7  update  acwnbr 

8  send  Initial  Parameters 

9  else  if  (Join  Request) 

10  send  Join  Request 

11  end 

12  else  !*  external  channel  ready  */ 

1 3  if  (message  originator  =  acwnbr) 

14  if  (Status  Report) 

1 5  send  Status  Report  to  MONITOR_PROCESS 

16  else  if  (Token) 

17  if  (Serial_Number  =  Expected_Serial_Nuniber  -  1) 

18  send  Token _Ack  /*  to  acwnbr  */ 

19  end 

20  if  (Serial_Number  =  Expected_Serial_Nuniber ) 

21  send  Token  to  AgreeProcessor 

22  send  Token  Ack  /*  to  acwnbr  */ 

23  increment  Expected_Serial_Number 

24  end  /*  out  of  order  messages  are  discarded  */ 

25  else  if  (Token  Pool)  /*  Token_Pool  is  always  accepted  */ 

26  send  Token  Pool  to  AgreeProcessor 

27  send  Token  Ack  /*  to  acwnbr  */ 

28  set  Expected_Serial_Number  =  Serial_Nuniber  +  1 

29  end 
|30  end 
31  end 

I _ _ _ _ — _ — - - 

Figure  10  FIFO  Channel  -  Back  Process 

E.  AGREEPROCESSOR 

The  specification  for  the  AgreeProcessor  was  rewritten  to  account  for  various 
subtleties  and  to  improve  overall  readability.  All  tokens  received  through  the  FIFO 
channel  layer  are  sent  to  AgreeProcessor  for  dispatching  to  the  appropriate  processor.  A 
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duplicate  token  is  one  that  has  been  previously  processed  at  given  member.  Some  scheme 
to  detect  and  reject  duplicate  tokens  is  required.  Proper  termination  of  the  agjree  phase 
and  subsequent  initiation  of  the  commit  phase  are  also  required.  Figure  11.  In  this 
section,  we  discuss  the  operation  of  the  agree  processor. 

AgreeProcessor  for  agreCpjip^)  at  p. 

1  if  (not  blocked  by  CommitProcessor) 

2  if  (initiate  agreement  message  received)  /*  p^  =  p^  */ 

3  add  agreepfp^)  to  TokenPoolip) 

4  STpJ(pi)  <-  joinagreed  or failagreed 

5  send  agreCpf^p^  to  cwnbr{p) 

6  send  acknowledgment  to  calling  process 

7  else  /*  a  token  or  external  token  pool  is  received  */ 

8  if  {ExtTokenPool) 

9  for  Vtokens  €  ExtTokenPool 

10  if  (token  €  TokenPoolip^)) 

11  if  (originator  failed) 

1 2  ProcessToken 

13  end 

14  else/*  token  not  in  TokenPool  */ 

1 5  if  (received  for  the  first  time) 

16  ProcessToken 

17  end 

18  end 

19  end 

20  else  /*  a  token  was  received  */ 

21  if  (received  for  the  first  time)  i 

22  ProcessToken  | 

23  end 

24  end 

25  end 

26  end 


Figure  11  Agreement  Processor 
1.  Initiate_Agreenient  Message 

When  AgreeProcessor  receives  an  initiate  jagreement  message,  the  appropriate 
agree  token  is  generated,  added  to  the  token  pool  and  forwarded  to  the  cwnbr.  The 
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status  table  entry  for  the  subject  is  updated.  An  acknowledgment  is  returned  to  the  calling 
process.  This  reflects  no  change  to  the  prior  specification. 

2.  External  Token  Pool 

When  an  external  token  pool  is  received,  it  is  compared  to  the  local  token  pool. 
All  tokens  in  the  external  token  pool  are  examined.  Processing  of  a  token  depends  on 
whether  the  token  is  also  present  in  the  local  token  pool. 

If  an  agree  token  from  the  external  token  pool  is  in  the  local  token  pool,  the 
token  originator  may  have  failed.  Due  to  a  failure  of  the  originator,  the  agree  token  is 
requires  conversion  into  a  commit  token  at  the  first  active  clockwise  neighbor  of  the 
originator  only.  Such  tokens  are  processed  as  if  they  had  been  received  as  a  separate 
token  message.  If  the  token  is  not  present,  it  may  have  already  been  purged.  These 

tokens  must  be  rejected  as  duplicates. 

a.  Token  Originator  Failed 

Detection  of  the  token  originator  failing  prior  to  initiating  the  commit 
phase  is  accomplished  separately  for  joinagree  mdfailagree  tokens.  Figure  12.  It  is 
necessary  for  the  next  active  member  to  detect  the  originator’s  failure  and  also  initiate  the 
commit  phase  for  the  incomplete  change  started  by  the  originator.  The  agree  token  may 
be  received  as  part  of  the  external  token  pool.  The  token  will  also  be  present  in  the  local 
token  pool  from  the  acwnbr  of  the  failed  originator.  This  situation  may  also  occur  if  a 
member  in  the  middle  of  the  ring  fails.  The  failed  member's  cwnbr  will  receive  an  external 
token  pool  containing  the  original  agree.  However,  this  does  not  require  a  commit  to  be 
initiated  as  the  originator  has  not  failed.  It  is  essential  that  the  differences  be  noted  and 
accounted  for.  The  same  conditions  are  present,  but  different  processing  must  occur. 

The  originator’s  failure  during  a  joinagree  phase  can  be  detected  if  the  rank 
of  the  external  token  pool  originator  is  greater  than  the  rank  of  the  current  member. 
Consider  the  host  failing  prior  to  initiating  the  commit  phase.  All  members  in  the  group 
have  agreed  to  the  join.  The  joinagree  token  will  be  received  by  the  new  host  upon  ring 
reconfiguration.  The  new  host's  rank  is  zero  (0)  while  the  external  token  pool  originator’s 
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rank  is  the  (group  size  -  1).  Since  Rankioh^nator)  >  Rankihost),  a  commit  must  be 
initiated. 


Consider  member  p,  with  rank  /  failing  during  the  joinagree  phase.  When 
the  failure  of  p,  is  detected,  p,^,  receives  the  external  token  pool  from  p,.;.  The  joinagree 
token  is  present  in  both  the  external  token  pool  and  TokenPooKp^^^).  The  token  is 
rejected  since  Rankip.,^)  <  Rank(p.^^), . 

LostAgreeToken 

1  if  (joinagree) 

2  if  (rankip)  >  rank(p)) 

3  return  true 

4  else 

5  return  false 

6  end 

7  end 

8  if  (faiiagree) 

9  if  (ReludveRankiPi^ ,  p.)  >  RelaHveRank(pj ,  p.)) 

10  return  true 

11  else 

12  return  false 

13  end 

14  end 


Figure  12  Determination  of  Token  Originator's  Failure 

Now  we  consider  a  duplicate  faiiagree  token.  Define  RelativeRankipj,  p,) 
as  the  rank  of  p^  with  respect  to  p.  instead  of  the  host.  A  ring  transversal  starts  and  ends  at 
the  same  specified  process,  i.e.  any  given  member  foUows  itself  in  a  ring  transversal. 
Figure  13.  RelativeRank  for  a  process  that  ic  not  a  member  of  the  group  is  undefined. 
Recall  the  subject  of  a  faiiagree  remains  a  member  of  the  group  view  until  the  commit  is 
processed.  A  lost  faiiagree  token  is  determined  by  the  RelativeRank  of  the  token  subject 
p,  and  token  pool  originator  p^. 
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Figure  13  Relative  Rank 


If  the  RelativeRankip,,  p,)  ^  RelativeRank{p^,  p.)  the  token  originator  has 
failed  and  the  faUcomU  phase  should  be  initiated.  This  situadon  will  occur  if  the  failed 
token  originator  itself  fails  prior  to  initiating  the  commit  phase. 
b.  DupUcate  Processing 

Conditions  to  detect  if  a  token  has  been  received  and  processed  already  are 
summarized  in  Table  1.  It  is  necessary  to  detect  duplicate  processing  if  the  external  token 
pool  contains  tokens  not  found  in  the  local  token  pool.  There  are  two  possible  ways  for 
this  to  occur.  If  the  token  has  not  been  received  and  processed,  or  the  token  has  been 
purged  from  the  local  token  pool. 


Table  1  CONDITIONS  TO  DETECT  DUPLICATE  PROCESSING 


Token 

CfiOditiflQ 

joinreqst 

joinagree 

joincomit 

P*  e  GV^ 

failagree 

failcomit 

P*  « 

Consider  the  commit  phase.  All  members  of  the  group  have  agreed  to  a 
particular  ring  reconfiguration.  Due  to  the  latency  of  token  transmission  around  the  ring, 
not  all  members  commit  the  change  simultaneously.  Recall  the  token  pool  is  purged  of  old 
commit  tokens  and  the  corresponding  agree  token  prior  to  the  commit  token 
transmission.  If  a  member  that  has  committed  a  change  receives  an  external  token  pool 
from  a  member  that  has  agreed  to  the  change,  the  agree  token  remains  in  the  external 
token  pool.  Likewise,  a  previous  commit  token  may  be  received  as  part  of  the  external 
token  pool.  Since  the  tokens  have  been  processed  and  removed  from  the  local  token  pool, 
it  is  necessary  to  check  the  effects  these  tokens  may  have  had  on  the  group  view,  had  they 
been  previously  processed.  For  duplicate  join  tokens,  the  subject  would  already  be  a  part 
of  the  group  view.  Conversely,  the  subject  of  duplicate  fail  tokens  would  have  already 
been  removed  from  the  group  view.  In  this  manner,  duplicate  processing  can  be  detected 
and  the  tokens  rejected. 

It  is  not  necessary  to  include  all  possible  conditions  for  a  token  having  been 
processed.  Recall  the  duplicate  processing  check  occurs  only  if  the  token  is  not  present  in 
the  local  token  pool.  For  a  Joinreqst,  if  the  token  is  received  as  pan  of  the  external  token 
pool  and  has  been  purged  from  the  local  token  pool,  the  Joincomit  must  have  occurred. 
Since  the  result  of  the  joincomit  is  subject  becoming  a  part  of  the  group,  it  is  only 
necessary  to  check  the  end  result  Intermediate  stages  of  the  join  process  have  not 
deleted  the  joinreqst  token  from  the  local  token  pool.  Since  the  token  remains  in  the 
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token  pool,  it  is  in  both  local  and  external  token  pools  and  is  rejected.  Similar  logic  results 
in  the  conditions  presented  in  Table  1. 

3.  Tokens 

Each  type  of  token  is  handled  individually  upon  receipt  by  AgreeProcessor. 
Figure  14.  AgreeProcessor  acts  as  a  filter  to  remove  duplicate  tokens  before  forwarding 

non-agree  tokens  to  the  appropriate  processor.  Agree  tokens  are  processed  locally. 

a.  Joinreqst  Tokens 

The  joinreqst  tokens  are  forwarded  to  JoinProcessor  for  further 

processing. 

b.  Agree  Tokens 

If  the  current  process  is  not  the  originator  of  the  token,  and  it  is  not  part  of 
the  local  token  pool,  it  is  added  to  the  token  pool,  the  status  updated  and  the  token  sent  to 
the  cwnbr.  This  accounts  for  the  first  time  an  agree  token  is  received  and  processed. 
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ProeessToken 

1  if  (joinreqst) 

2  send  token  to  JoinProcessor 

3  elseif  {commit) 

4  send  token  to  ComitProcessor 

5  dseif  (agree) 

6  if  ((p.  *  Py)  &&  (agree  token  e  TokenPool(p)) 

I  add  agreepj<Pt)  to  TokenPool(p) 

8  ^^ft(P/k)  ^  FailAgreed  or  JoinAgreed 

9  send  agrecpfp^)  to  cwnbrip) 

10  else  Py 

I I  if  ((p.  =  py)  II  (Vp,  I  p,^p.,  p,  €  ST pj)) 

12  compute  rank  Vp|€  STp^  with  Agreed  status 

1 3  if  rank(pj)  =  smallest 

14  send  initiate.comit  to  ComitProcessor 

15  else 

16  *-joinpendg  orfailpendg 

17  end 

18  end 

19  end 

20  end 

Figure  14  Processing  Agree  Tokens 

If  the  current  process  is  the  token  originator,  or  the  first  active  process 
clockwise  fix)m  the  originator  that  receives  the  agree  token  after  it  circulates  around  the 
ring,  the  rank  of  all  processes  with  an  agreed  status  is  computed.  An  initiate  jcommit  is 
transmitted  if  the  subject  is  the  lowest  ranked  of  all  processes  fulfilling  the  above 

conditions.  If  not,  the  subject's  status  is  updated  to  pending, 
c.  Commit  Tokens 

A  commit  token  is  immediately  sent  to  CommitProcessor  for  processing. 

4.  Side  Effects 

The  external  tt^en  pool  mentioned  above  requires  a  new  message  type, 
Ej^TknPool.  Figure  15.  The  message  includes  the  originator  of  the  token  pool  as  an 
internal  field.  This  is  required  by  AgreeProcessor  for  determination  of  a  failed  token 
originator. 
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EXTERNAL  FORMAT 


SERIAL 

NUMBER 


Ni 


MESSAGE 

ORIGINATOR 


rn 


MISaSAOS 

OftieBCAtHMd 


Ni 


POOL 

SIZE 


TOKEN 


TOKEN 


# 


TOKEN  FIELD 


sp  =  Space  character 

TOKEN 

sp 

TOKEN 

r 

TOKEN 

Vi  =  New  line  character 

TYPE 

SUBJECT 

n 

ORIGINATOR 

Figure  15  External  Token  Pool  Message  Foimat 


F.  MULTIPLE  JOINS 

1.  Problem 

Consider  two  members  attempting  to  join  a  group  almost  simultaneously. 
Figures  16-18.  The  first  member's  join  will  be  completed  properly.  Upon  completion  of 
the  first  join^  it  is  possible  to  begin  processing  the  second  member's  Joinreqst  before  the 
FIFO  channels  reflect  the  re-configured  ring  with  the  first  member  in  it.  It  is  possible  to 
complete  the  second  join  before  the  channels  are  re-configured.  This  can  happen  because 
of  the  de-coupled  protocol  and  FIFO  channels.  The  host  does  not  change  its  acwnbr  until 
it  initiates  a  status  query  to  the  last  member.  The  FIFO  channel  determines  a  member's 
acwnbr  as  the  target  of  the  most  recent  status  jry.  The  cwnbr  is  the  originator  of  the 
most  recent  statusqry  received.  There  is  an  inherent  latency  involved  in  the  FIFO  channel 
reconfiguration  due  to  the  timing  considerations  of  subsequent  statusqrys.  Thus,  it  is 
possible  for  a  joinreqst  from  a  second  new  member  to  complete  both  phases  of  theyoin 
process  prior  to  FIFO  channel  reconfiguration.  The  first  new  member  may  never  have 
processed  the  joinagree  and  joincomit  for  the  second  joining  member  prior  to  the  second 
member  being  incorporated  into  the  group  view.  When  the  first  joining  member 
determines  its  acwnbr  and  receives  the  external  token  pool,  the  joincomit  token  for  the 
second  member  may  be  present.  Processing  the  token  will  result  in  attempts  to  delete  the 
corresponding  nonexistent  joinagree  token  never  received  by  the  first  member  and 
subsequent  removal  of  the  second  member  from  the  status  table.  Both  of  these  are  error 
conditions. 
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Figure  16  Simultaneous  Joins 


2.  Solution 

Initiate  the  FIFO  reconfiguration  upon  transmission  of  the  initial  parameters  to 
the  new  member.  Figure  10,  line  7. 


Figure  17  Group  View  after  1  member  has  joined 
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3.  Justification 


The  FIFO  channel  can  be  considered  to  be  re-configured  when  the  host 
determines  its  acwnbr.  The  response  from  the  new  acwnbr  is  not  required,  as  the  FIFO 
channel  reject  all  tokens  unless  they  are  sent  by  the  acwnbr.  Tokens  from  the  old  acwnbr 
are  rejected  and  must  be  processed  by  the  new  acwnbr  prior  to  being  forwarded  to  the 


host  member. 


Figure  18  Group  View  at  Host  &  J2 


G.  OTHER  IMPROVEMENTS 

In  this  section,  oversights  to  the  protocol  specification  and  the  modifications  required 
are  briefly  described, 

1.  Joinreqst  Token  Processing 
a.  Problem 

When  processing  a  join  message  from  a  prospective  member,  Initiatejoin 
adds  a  joinreqst  token  to  the  token  pool  prior  to  generating  it. 
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b.  Solution 

If  the  current  message  being  processed  is  a  join  message,  generate  the 
joinreqst  token  and  then  add  the  token  to  the  token  pool.  Figure  19,  lines  10-13. 
Initiatejoin  for  a  join  request  message/token  for  p^  at  P; 

1  while  (true) 

2  if(p«„«5r„.GV„) 

3  receive  join  request  message  or  token  for  p^ 

4  end 

5  if  (Pi  =  P*„«) 

6  send  initiate  agreement  message  to  AgreeProcessor  for  p^ 

7  block  until  AgreeProcessor  acknowledges  end  of  processing 

8  else 

9  <-  JoirJiequested 

10  if  (join  request  message)  /*  p^  locates  p,  and  sends  its  join  request  */ 

1 1  generate  joinreqff?^  token 

12  end 

13  add  joinreq^^ip^)  to  TokenPool(p) 

14  send  joinreq  token  to  cwrdnip) 

15  end 

16  end 


Figure  19  Processing  of  a  Join  Request  Message  /  Token 

2.  Commit  Token  Generation 

a.  Problem 

If  a  member  was  in  a  pending  status,  a  commit  token  for  that  process  was 
never  generated  prior  to  committing  the  change.  These  members  would  remain  pending 

indefinitely. 

b.  Solution 

Create  the  commit  token  before  committing  the  change  for  a  member  with 
a  pending  status.  Figure  20,  lines  11-12. 
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ProcessCommitTkn  for  commitpjip^)  at 

1  if  (initiate  commit  message  received) 

2  generate  commit  token 

3  token  to  be  processed  generated  token 

4  else  if  ((p^  ^  pp  &&  (not  duplicate)) 

5  token  to  be  processed  <-  received  token 

6  else 

7  exit 

8  end 

9  CommitChange 

10  while  ( p,  e  STp^  with  pending  status  &  Rank(p,)  <  Rank(p  J,  p^  e  STp) 

1 1  generate  commit  token 

12  token  to  be  processed  f-  generated  token 

1 3  CommitChange  in  rank  order 

14  end 


Figure  20  Generate  /  Receive  and  Process  a  Commit  Token 

3.  Message  Queue  in  the  FIFO  Channel  Layer 

The  front  processor  was  modified  to  transmit  the  head  of  the  message  queue 
after  receiving  any  message,  either  on  the  internal  or  external  channel.  Figure  21. 
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FIFO  Channel  -  FRONT  Process 

1  Wait  for  a  chani.el  ready  to  read 

2  if  (external  channel  ready) 

3  if  (Status  Query) 

4  send  Status  Query  to  MonitorProcess 

5  else  if  (JoirF.equest) 

6  send  JoinRequest  to  JoinProcessor 

7  else  if  (InitialParameters) 

8  send  InitialParameters  to  JoinProcessor 

9  else  if  (TokenAck) 

10  if  (Received_Serial_Number  =  Expected_serial_nuinber) 

1 1  remove  Head_of_Queue 

12  deaement  Queue_Counter 

13  end 

14  end 

15  else  /*  internal  channel  ready  */ 

16  if  (Token) 

17  change  Token  to  external  format  /*  add  external  header  */ 

1 8  insert  Token  in  queue 

19  increment  Serial_Number 

20  increment  Queue_Counter 

21  else  if  (TokenPoof) 

22  discard  all  messages  in  queue 

23  change  TokenPool  to  external  format  /*  add  external  header  */ 

24  insert  TokenPool  in  queue 

25  increment  Serial_Number 

26  increment  Queue_Counter 

27  else  if  (StatusReport) 

28  update  cwnbr 

29  send  StatusReport  to  cwnbr 

30  end 

31  end 

32  if  (Queue_Counter  >  0) 

33  send  Head_of_Queue  to  cwnlw 

34  set  Expected_serial_nuniber  =  Head_of_Queue_serial_number 

35  end 


Figure  21  FIFO  Channel  -  Front  Process 
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H.  SYNOPSIS 


This  chapter  has  described  the  changes  that  were  required  to  successfully  implement 
the  group  membership  protocol.  Changes  covered  coding  as  well  as  protocol  related 
problems  not  discovered  in  the  original  specification.  These  changes  deal  with  the  correct 
functioning  of  the  protocol  and  do  not  address  performance  issues.  Performance  is  dealt 
with  in  Chapter  IV. 
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IV.  PERFORMANCE  OF  THE  GMP 


The  performance  of  the  protocol  on  the  Electrical  and  Computer  Engineering  Local 
Area  Network  (ECE  LAN)  consisting  of  SUN2  workstations  connected  via  an  Ethernet, 
was  measured  and  the  results  are  presented  in  this  chapter. 

A.  LATENCY 

The  latency  involved  in  processing  changes  to  the  group  view  is  measured  by  each 
member  using  the  local  time  clock  on  each  specific  processor.  Timestamps  were 
generated  for  the  conditions  listed  in  Table  2. 


Table  2  CONDITIONS  WARRANTING  A  TIME  STAMP 


Time  Stamp 

Where  Taken  in  Code 

L 

initiate  an  agree  token 

receive  an  agree  token 

tc. 

send  a  commit  token 

The  timestamps  and  related  data  were  dumped  to  a  file  local  to  each  processor.  A 
filter  program  was  designed  and  written  to  compile  the  data  from  the  various  systems, 
given  a  list  of  the  members  in  the  test  group,  and  the  maximum  number  of  members.  The 
filter  program  was  designed  for  a  restricted  set  of  all  possible  group  views  given  the  above 
data.  The  case  when  the  group  view  starts  as  the  initial  member,  grows  to  the  maximum 
number  of  members  and  then  shrinks  to  the  original  host  is  the  only  possible  case  handled 
by  the  filter.  Figure  22  illustrates  the  required  group  view  changes. 
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Figure  22  Group  Changes  Required  by  the  Filter  Program 


A  further  restriction  is  that  all  members  departing  the  group  must  be  the  host's 
acwnbr.  This  removes  the  need  for  a  local  database  in  the  filter  program  to  track  which 
members  are  part  of  the  current  iteration  of  timestamp  evaluations.  Future  improvements 
to  the  filter  program  can  implement  a  dynamic  database  to  account  for  all  possible  changes 
to  the  test  group.  The  format  of  the  output  file  is  shown  in  Figure  23. 


agree 

B3 

sec 

eg 

usee 

init 

agree 

E3 

sec 

sp 

usee 

reev 

comit 

sec 

B3 

usee 

send 

sp  =  space 

_ 

Figure  23  Time  Stamp  File  Format 

The  time  required  to  implement  a  change  at  a  given  member  i  is  calculated  by 

subtracting  the  initial  time  stamp  from  the  completion  time  stamp. 

ti  =  tcs  host 

ti=tcs-tai  I  i  =  h0St 

The  average  time  to  commit  a  change  at  each  member  was  calculated  as  follows; 

fi 

Sr. 


r  = 


n 


However,  as  the  group  increases  from  n  members  to  n+7  members,  ihtjoinagree 
must  be  processed  by  the  n  members  currently  in  the  group.  Similarly,  the  n  members 
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must  receive  and  process  the  joincomit  token.  Hence,  for  a  join  to  a  group  of  n  members, 
the  processing  time  must  be  n{tagree+tcomu),  where  is  the  time  required  to  process  an 
agree  token  and  is  the  time  to  process  a  commit  token.  The  communication  time  to 

transmit  the  token  from  one  member  to  its  neighbor  is  Since  each  of  the  tokens  must 
be  transmitted  to  the  n  members,  the  total  communication  cost  is  t  =  2n  *  (tcomm)  ■ 
Therefore,  the  total  time  required  to  implement  a  join  at  all  members  of  a  /i  member  group 

is 

t  —  ft{tagree  t  commit  '2-tcomm') 

Notice  that  the  time  to  implement  a  change  is  proportional  to  the  size  of  the  group. 


Table  4  PROCESSING  TIME  VALUES 


Time 

Occurrence 

time  to  process  an  agree  token 

^commit 

time  to  process  a  commit  token 

^comm 

inter-member  communication  time 

Similarly,  the  expected  time  for  a  failure  can  be  determined  to  have  a  linear 
relationship  to  the  size  of  the  remaining  group. 

B.  TESTING 

The  performance  of  the  GMP  was  tested  on  the  ECE  LAN  consisting  of  SUN 
workstations  linked  via  an  Ethernet.  There  were  no  gateways  between  any  of  the 
members  of  the  group.  Only  single  complete  changes  were  allowed  at  any  given  time.  A 
complete  reconfiguration  included  the  underlying  FIFO  channel  as  well  as  the  logical  ring 
structure.  A  linear  relationship  was  observed  between  the  number  of  members  in  the 
group  and  the  average  time  it  took  to  commit  the  new  member.  Figure  24.  Since  the 
communication  depends  heavily  upon  the  network  load  as  well  as  the  individual  processcn* 
load,  average  values  over  a  large  variety  of  conditions  such  as  time  of  day  and  number  of 
people  on  the  network  were  generated  in  order  to  get  reliable  data  points. 
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Figure  24  Average  Time  for  each  Member  to  Implement  a  Join  to  the  Group 

Similarly,  the  time  required  to  remove  a  member  from  the  group  view  was  obtained  and 
plotted.  Thus,  the  relationship  between  time  and  group  size  was  determined  for  a 
decreasing  group  size.  Again,  only  single  con:q)lete  changes  were  allowed.  Figure  25.  A 


linear  relationship  was  observed  as  expected. 


Figure  25  Average  Time  to  Implement  a  Failure  in  the  Group 
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V.  METHODS  FOR  IMPROVING  PERFORMANCE 


In  this  chapter  we  explore  two  methods  that  might  be  used  to  further  improve  the 
overall  performance  of  the  protocol.  Each  could  be  incorporated  along  with  the  other  or 
individually. 

A.  MESSAGE  REDUCTION 

The  protocol  as  currently  implemented  has  a  high  overhead  due  to  the  number  of 
inter-member  messages  required  to  effect  changes  in  the  logical  ring  structure.  In  order  to 
reduce  the  number  of  messages  required  for  the  maintenance  of  the  logical  ring,  structure 
three  methods  are  discussed  below. 

1.  TokenPool  versus  Tokens 

Consider  multiple  near  simultaneous  changes  to  the  group  view.  Transmitting 
the  token  pool  instead  of  individual  tokens  will  result  in  a  reduction  of  messages  if  the 
changes  occur  close  enough  together  such  that  the  token  pool  for  one  change  includes  the 
tokens  for  the  subsequent  changes.  This  will  result  in  a  decreased  number  of  messages. 
However,  the  probability  of  such  changes  occurring  is  minimal.  Since  changes  to  the 
group  are  uncorrelated,  the  probability  of  such  changes  occurring  is  minimal.  The 
corresponding  reduction  of  message  traffic  is  negligible.  Additionally,  there  is  an  increase 
in  the  size  of  the  message  for  most  traffic.  Modifications  required  to  effect  this  include  the 
dynamic  generation  of  the  group  view  by  the  FIFO  channel.  Instead  of  receiving  the  token 
pool  for  generation,  the  FIFO  channel  would  receive  a  flag  and,  at  that  point,  generate  the 
external  token  pool  message.  One  such  method  might  be  to  set  a  flag  that  indicates  that 
the  token  pool  must  be  transmitted.  FIFO  properties  are  maintained  by  the  order  in  which 
the  tokens  are  processed  upon  receipt.  Reduction  will  occur  only  if  multiple  changes 
occur  prior  to  the  transmission  of  the  token  pool  for  the  initial  change.  Problems  arise  in 
the  correct  setting  of  the  transmit  flag;  i.e.,  did  change  #2  get  sent  in  the  last  token  pool. 
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or  is  another  token  pool  transmit  required.  This  method  is  not  recommended  since  the 
number  of  messages  is  reduced  only  in  special  cases. 

2.  Periodic  Token  Pool 

Recall  that  tokens  are  generated  only  if  a  change  to  the  group  view  occurs. 
Another  method  that  will  reduce  the  number  of  inter-member  messages  is  to  periodically 
send  the  token  pool  instead  of  individual  tcdcens.  In  this  manner,  multiple  tokens  can  be 
transmitted  simultaneously.  Message  reduction  is  indicated  only  if  multiple  changes  to  the 
group  view  occur  within  the  period  of  the  token  pool  transmission.  If  this  does  not  occur, 
message  traffic  will  actually  increase;  i.e.,  if  there  are  no  changes  within  this  period  the 
token  pool  is  still  transmitted.  This  method  will  also  increase  the  latency  in  phase 
completion  as  tokens  are  not  immediately  forwarded  around  the  ring.  Additionally,  the 
message  size  will  be  increased.  This  method  is  not  recommended  either. 

3.  Piggyback  the  Token  Pool 

A  further  refinement  would  be  to  include  the  local  token  pool  as  part  of  the 
status  report.  The  monitoring  member,  upon  receipt  of  a  statmrpt,  would  parse  the  token 
pool  and  process  the  appropriate  tokens.  FIFO  channel  requirements  are  maintained  by 
the  order  in  which  tokens  are  processed  upon  receipt  of  a  token  pool.  This,  however, 
leads  to  additional  processing  for  every  status  report. 

Difficulties  might  occur  in  the  latency  of  cycle  completion  in  a  large  group. 
Most  notably,  consider  when  a  new  member  has  requested  to  join  an  existing  group. 
Define  r,  as  the  latency  within  a  process.  It  is  the  difference  in  time  between  receiving  the 
token  pool  via  a  statusrpt  and  transmitting  the  tokens  around  the  ring,  can  be  modeled 

as  a  random  variable.  Ignoring  the  communication  time,  t _ _  and  the  processing  time, 

associated  with  the  normal  processing  of  tokens,  the  latency  for  a  change  to  an  N 

member  group  (i.e.  a  join  request)  becomes: 

Ttolal  —  2(jN  X  tl} 

Thus,  the  latency  involved  may  become  prohibitive  for  large  N.  This  method  is 
recommended  for  implementation,  provided  that  the  latency  of  changes  is  not  important. 
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Care  must  be  taken  to  ensure  the  time-out  on  a  join  request  is  large  enough  to  encompass 
the  worst  case  scenario.  However,  since  the  time-out  is  finite,  this  method  place  an 
implicit  upper  bound  on  the  maximum  group  size.  Once  the  group  is  large  enough,  new 
members  will  be  unable  to  join  due  to  the  time  needed  to  implement  the  join  around  the 


ring. 

B.  SINGLE-THREADED  PROGRAM 

1.  Problem 

The  current  design  of  the  protocol  involves  concurrent  processes  handling 
specific  areas  of  responsibility.  Fully  implementing  this  design  would  allow  each  process 
to  reside  on  different  processors.  However,  in  most  cases,  a  single  processor  is  the  norm. 
There  is  a  significant  amount  of  overhead  due  to  the  context  switching  and  intra-member 
messages.  The  asynchronous  nature  of  the  protocol  has  several  areas  requiring  process 
blocking  as  mentioned  in  chapter  m. 

2.  Solution 

Redesign  the  main  process  using  a  single-threaded  program  .  Figure  26  shows 
the  recommended  processes  and  inter-dependencies. 


Figure  26  Single-Threaded  Process  Inter-E)ependencies 


3.  Justification 

The  TokenProcessor  would  be  a  single-threaded  program  combining  all  aspects 
of  the  current  design  as  shown  in  Table  4.  The  watchdog  timer  must  still  be  a  separate 
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entity  by  definition.  The  FIFO  channel  is  not  included  in  the  TokenProcessor  as  Back  and 
Front,  are  by  nature,  separate  programs  without  any  timing  restrictions. 


Table  4  SINGLE  THREADED  PROCESSES  AND  EQUIVALENTS 


Single-Threaded  GMP 

Timer 

Timer 

JoinProcessor 

AgreeProcessor 

ComitProcessor 

IntegrateAfember 

TokenProcessor 

Status  Table  Manager 

Group  View  Manager 

Token  Pool  Manager 

StatusMonitor 

StatusReporter 

BACK 

BACK 

FRONT 

FRONT 

The  asynchronous  nature  of  the  design  would  be  eliminated.  The  need  for  block 
and  wait  would  be  eliminated  if  a  single-threaded  program  design  were  to  be  used. 
Additionally,  the  design  would  result  in  a  significant  decrease  in  the  overhead  costs  due  to 
the  inter-process  messages  and  connecting  services  being  eliminated.  The  single-threaded 
program  also  elinunates  the  need  for  separate  database  managers.  The  TokenProcessor 
can  maintain  all  databases  internally,  with  different  pointers  keeping  the  different  databases 
separate. 

Concurrent  with  the  new  design,  a  review  of  all  subroutines  is  recommended. 
Current  design  and  programnting  practices  preserves  all  data  passed  to  the  subroutines. 
This  can  lead  to  signiHcant  overhead  since  the  data  is  stored  multiple  times. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


In  this  thesis,  the  modifications  required  to  implement  the  group  membership 
protocol  as  proposed  by  [5]  are  presented.  The  protocol  has  been  successfully 
implemented  and  to  date  has  run  continuously  for  more  than  48  hours  with  a  stable  group 
membership.  Additionally,  a  group  size  of  20  members  was  achieved.  These  results, 
though  preliminary,  are  the  first  for  this  protocol.  Although  the  protocol  is  functioning, 
continued  debugging  and  improvement  are  currently  going  on. 

As  expected,  the  time  required  to  implement  a  change  was  found  to  have  a  linear 
relationship  to  the  eventual  group  size. 

Further  work  should  include  the  re-design  of  the  protocol  as  a  single-threaded 
program.  In  this  manner,  the  response  of  the  protocol  can  be  improved  as  inter-process 
communication  time  is  reduced  drastically.  However,  attempting  to  implement  the 
message  reduction  schemes  to  improve  performance  is  not  recommended  as  there  is  little 
to  gain  in  the  number  of  messages.  On  the  contrary,  implementation  of  the  message 
reduction  schemes  would  result  in  a  large  increase  in  the  latency  of  changes  to  the 
membership. 

Additional  research  is  suggested  in  the  area  of  network  partitioning.  Consider  that  a 
network  may  partition  in  two  separate  halves  that  are  fully  connected  on  either  side  of  the 
boundary.  A  group  originally  existing  on  both  sides  will  become  two  groups  with  the 
same  name  operating  independently  on  either  side  of  the  partition.  The  difficulty  arises 
when  the  network  is  repaired.  The  protocol  does  not  provide  for  the  possibility  of 
merging  the  two  groups  back  into  the  original  group.  The  problem  of  handling  network 
partitioning  is  non-trivial. 
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APPENDIX 

The  following  code  is  included  for  completeness. 

TABLE  OF  CONTENTS 

Definitions  &  Utilities  .  42 

Simple  Application  &  Main  Process  .  85 

FIFO  Channel  Processes  .  95 

Monitor  Processes  .  105 

Agree  Processor  .  116 

Commit  Processor  .  131 

Integrate  Member  Process  .  144 

Join  Processor  .  151 

Database  Managers  .  159 

Data  Cruncher  Program .  180 
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\0  •- 


STARTTIMR 

ECLTIOKEN 


Table  of  Contents 


FIFO  Utilities  .  45 

enqueue  .  45 

dequeue  .  46 

get_queue_head  .  46 

flush_queue  .  47 

send_msg_fTont  .  47 

send_nisg_back  .  48 

send_ack  .  48 

send_msg_in  .  49 


44 


nove  a  msg  from  the  head  of  the  queue 'qup^  get_queue_head-feaims  a  pointer  to  the  message  at  the  head  of  the  queue. 


^4i4»lt4t4i4i*4t4i**4t4i4t>Mt4(*4t4t4(4(4c4t4(4c4c4c4t4t4t4i4(*4(4c4i4K4t**4ntc4i4c4(4(4t>|t4(*4c4(4t4t4(4(3|c*4(*>|( 

flushjiueie-iefnoveaU  nodes  of 'qu|]|i' Grom  menK^  send_insg_fiatt- sends  an  external  message  ^nisg' to  the  front  ported  the 

AB  iBcd  memory  is  dealkxaMed,  qxciiied  IP  destination 'dest'.  'dest' is  a  string  with  efementadikessfonnaL 


(g(msg,  strien(msg),  IPaddr,  port); 


if  ( (wrilemsg(sockf(l,  iiiinsg,  msgfenX 
to  the  prindrsend_insg_iii:  write  enor  on 


to  server*/ 


Table  of  Contents 


GMPUtiUties  .  51 

GetAcwnbr  .  52 

InStatusTable  .  55 

CountDown  .  56 

SendTkn2Agr  .  56 

InGroup  .  57 

GetMembWithRank  .  58 

InTokenPool  .  59 

TokensREqual  .  60 

GetRank  .  61 

GetTokenType  .  62 

GetGroupSize  .  62 

GetStatus  .  63 

ReladveRank  .  63 

GetGroupView .  64 

GetStatusTable .  64 

GetTokenPool  .  65 

agreetoken  .  65 

committoken  .  65 

first_time .  66 
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GetAcwnbr  leiuns  0  if  acvvnlr  is  ccnectly  initialized  to  a  member 

address  as  ctxnpulea  fion  the  cunent  grcx^)  view,  obtained  bon 
the  group  view  manner  at  gvmsoc  and  the  current  status  table 
obtained  bom  stmsoc  according  to  the  rule  in  the  paper.  In  all 


gvbr^msglen]  = 


Seaich  f(T  myadck  in  gv  aiKl  lebm  ihe  element  ixecedi^ 
(modulo  ginq>  size)  that  is  NOT  pat  of  the  status  laUe. 


♦  #  # 


I 


■VA 

■y.i 

y{ 

■y.i 

Ai 

AK 

■AA 


O 

+ 


=  FALSE;  /"  found  the  Acwnbr  ♦/ 

icompieie= FALSE;  /*  finished  looking  but  did  not  find  one  V 


sgvsize+GVLl! 


JSTOFFSET;  /*fira  member  in  st  is  the  third  field  */ 


CouitDown:  Decictnenis  the  location  at  va^  after  at  least  Sen(fnai2Agn  A  token  is  sent  to  the  agreement  pnicess  and  the  caller  blocks 


ckse(agifd); 


3?^ 


1 

4. 


y*******************.*,***************************************************  if  ( gedhxnlist(gvlist,  &gvsiasttng,  GVLISTOFFSET  - 1)  =  0)  { 

GetMembWiihRaik:  retums  0  and  member  is  initialized  to  the  element  with  rank.  printf('\GetMembWithRank:  gv  parsing  for  gv  size  failed.'n"); 

-1  if  no  member  with  the  given  rank  exists.  printf(’NGetMembWithRank;  gv  =  l%sln ",  gvbuO; 

inember’ is  the  address  of  the  fust  character  of  the  member  string  printfCWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSNi"); 

to  be  returned.  Storage  is  allocaied  for 'member' which  should  be  exit(-l);) 


/•  Conven  the  gv  siring  to  a  list  *! 


Gen'okenType:  does  not  distuib  the  token  passed  ia  GelGnx^ize:ietunis  the  gioiq)  view  size.  Requires  the  caller  to  supply  a 

returns -I  if  the  token  type  is  invalid.  striitg  containing  the  group  view. 

inlGetTokenType(tQken)  intGetGroi^)Size(gv) 


)  rendGefTokcnType*/ 


lolcen.Qpe = GetToken'IVpe(tokBn); 


/*  get  ihe  token  type  */ 
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str21ist  .  68 

list2str  .  69 

removelist  .  70 

getfromlist  .  70 

listsize  .  71 

int_2_ext  . .  71 

get_sr_nbr  .  72 

msg_type  .  73 

in_nisg_type  .  74 

ext_msg_type  .  74 

get_target  .  75 

get_originator  .  75 

get_ext_target .  76 
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i»**»<i»»**»»»»***»<t»»***»**»»»**iii**»»»»*»»»**<i»»»»*  j*im******nt***********m***m**mm*****‘**i:tntinL7ttm<***m*ift‘****m**‘*****’if*******^t 

AUXILIARY  FUNCTIONS  (m^idiLc)  sti21ist- parse  a  string,  oeatiiig  a  list  of  nodes,  each 

of  which  points  to  a  field  of  the  original  shine. 


iifriC'liext; ): 


/.jqu  *  taSfmj  { 


«»*«***«««****«**«««**<i>***«*>i>*4»k*«»*4>****«'*>  if  (sircny(iype,"st 

msKtyi)e=STA' 


in_msg_type- extracts  ihe  type  tkid  of  a  message  that  ext_m^type- extracts  the  type  field  ofa  message  that 


get.iaiget- extracts  the  desdnaikm  field  of  an  get_ariginaitr-extractstheoriginatarfieldfianithe 


leain(nbr);  1  /*  end  gcLoriginalor  */ 

)  rendgel_la»get*/ 


/«i3gKi  1X3  taSpte  j  { 
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readmsg  .  81 

writemsg  .  82 

senmsg  .  83 

recmsg  .  84 
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[sin)); 


^»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»» 

an  Unix  Domain  socket  for  a  server  conneaUN  -  esnMishan  Unix  Domain  socket  for  a  client 

im  caiiectU>KserverjBlh) 


)  TendaealBUN*/ 


=  NULL;  f*  substitute  NULL  for lo  end  string  */ 


2  a-S* 

liil 


-a  J 
Ka 


^iSla 


*  I 

I 

C  cl 

I  ^ 

1  ^ 


m^5 


rlH!  I 


)  raid 


fCsenmsg:  can't  gel  ll%sl  host  enliyn",  IPatklr): 


lecinsg- reads  ai  external  mess^e'msg' at  the  /*'aUocaie  space  for  entire  message*/ 

specified  IP  socket  msgbuf=CALLOC(msglen  +  HEADERSlZE+ l.char); 

The  message  is  atomically  received,  and  is  striped 


if  ((msglen  =  aioKmsghead)) = 0) 


A  SIMPLE  APPLICATION 
and 

MAIN  PROCESS 

Simple  Application .  86 

Main  Process  .  88 
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piinlfC'Ennr  executing  finnNi"); 
printfCHr7$$$$$$$$$$$$$$$$$$ 
e*il(l):  1 


/*■  wait  until  one  pnxess  exits  V 
retun^  =  wait(  (int*)  NULL ); 


printfC’MAIN  H<3CESS: 


FIFO  CHANNEL 


FIFO  Channel  Process  Dependencies  . 96 

Front  Process  Specification  . 97 

Back  Process  Specification  . 98 

Front  . 99 

Back  .  102 
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Figure  A I  FIFO  (Zhannel  -  Process  Dependencies 
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FIFO  Channel  -  FRONT  Process 

1  Wait  for  a  channel  ready  to  read 

2  if  (external  channel  ready) 

3  if  (Status  Query) 

4  send  Status  Query  to  MonitorProcess 

5  else  if  (JoinRequest) 

6  send  JoinRequest  to  JoinProcessor 

7  else  if  (InitialParameters) 

8  send  InitialParameters  to  JoinProcessor 

9  else  if  (TokenAck) 

10  if  (Received_Serial_Number  =  Expected_serial_nuinber) 

1 1  remove  Head_of_Queue 

12  decrement  Queue_Counter 

13  end 

14  end 

1 5  else  /*  internal  channel  ready  */ 

16  \i  (Token) 

17  change  Token  to  external  format  /*  add  external  header  */ 

1 8  insert  Token  in  queue 

19  increment  Serial_Nuniber 

20  increment  Queue^Counter 

21  else  if  (TokenPool) 

22  discard  all  messages  in  queue 

23  change  TokenPool  to  external  format  /*  add  external  header  */ 

24  insert  TokenPool  in  queue 

25  increment  Serial^Number 

26  increment  Queue_Counter 

27  else  if  (StatusReport) 

2&  update  cwnbr 

29  send  StatusReport  to  cwnlnr 

30  end 

31  end 

32  if  (Queue_Counter  >  0) 

33  send  Head_of_Queue  to  cwnbr 

34  set  Expected  serial  number  =  Head  of  Queue  serial_number 

35  end 


Figure  A2  FIFO  Channel  -  Front  Process 
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FIFO  Channel  -  BACK  process 


1  Wait  for  a  channel  ready  to  ready 

2  if  (internal  channel  ready) 

3  if  {Status  Query) 

4  update  acwnbr 

5  send  StatusQuery 

6  else  if  {Initial  Parameters) 

7  update  acwnbr 

8  send  InitialParameters 

9  else  if  {Join  Request) 

10  send  Join  Request 

11  end 

12  else  /*  external  channel  ready  */ 

1 3  if  (message  originator  =  acwnbr) 

14  if  {Status  Report) 

1 5  send  Status  Report  to  MONrrOR_PROCES  S 

16  else  if  {Token) 

17  if  (Serial_Nuniber  =  Expected_Serial_Number  -  1) 

18  send  Token  Ack  I*  to  acwnbr  */ 

19  end 

20  if  (Serial^Number  =  Expected_Serial_Nuniber ) 

2 1  send  Token  to  AgreeProcessor 

22  send  Token  Ack  /*  to  acwnbr  */ 

23  increment  Expected_Serial_Nuniber 

24  end  /*  out  of  order  messages  are  discarded  */ 

25  else  if  {TokenJPool)  I*  Token_Pool  is  always  accepted  */ 

26  send  Token  Pool  to  AgreeProcessor 

27  send  Token  Ack  /*  to  acwnbr  */ 

28  set  Expected_Serial_Nuniber  =  Serial_Nuniber  +  1 

29  end 

30  end 

31  end 

Figure  A3  FIFO  Channel  -  Back  Process 
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•FROfT  PORT  MANAGER  *  primfT'Usagerfiontsockuc^soclammy.acidrstnepjoinpintinbi^''); 


)  /*'eiKlifdienFD_ISSET(sockun,&fdrcad)*/ 


/.IpIIMSpIBj  I 


printfTFRONT.  iivabd  message  type  l%(yn",msg:type): 


)  I*  end  if  (FD_lSSEr(sockun,  &fdread))  */ 


myjaddr.acwnbr); 


MONITOR  PROCESS 


Monitor  Process  Dependencies .  106 

Status  Reporter  Process  Specification  .  106 

Status  Monitor  .  107 

Satus  Reporter .  110 

Timer .  113 
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Figure  A4  Monitor  Process  -  Internal  Structure  and  Dependencies 


ReportStatus  process  at  p, 

1  if  (not  blocked  by  IntegrateMember) 

2  if  (querying  member  e  GVpj  or  has  joinagree  status) 

3  p^,  =  querying  member 

4  send  status  to  p^, 

5  if  (previous  querying  member  =  p^,) 

6  send  TokenPool{p)  to  p^„ 

7  end 

8  end 

9  end 

end  ReportStatus 


Figure  AS  Reporting  of  Status 
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clen=sizeof(caIler_addr);  )  /*  end  switch*/ 

)  /•end  while  TRUE*/ 
)  /*endmain*/ 


pnntfCHUsage:  StatusRepoiter  soc  myaddr  tpmsK  bontsocV); 


quay_components = s&2Iist(queiydata, 


t,l)  =  0)(  if(strcinp(mon,prev_mon) !=0)  { /*ifthercisanew 

ponent  parsing  faikri.'o");  sircpy(prev_mon,  mon);  /*  update  the  prcvic 


hee(lxiO;  /^storage  acquired  on  read  of  quay  retumed”'/  1  Erontfd-conneclUNffrontsoc); 


if  ( wrileniSgCfinntfid,  msg,  m^glen)  <  in^glen  ){ 
printfTVlSlaaisRepater  token  pool  send  failed.Nn'O; 


ilistCm^list,  &timennsg;type,  2) = 0)  { 

C  VTimer  msgtype  for  tqry  parsing  guiedV); 


>»♦»»»»»»»»»»»♦♦»♦»»»»»♦»♦♦»»»»»»»»»»»»»»»»»»»»»»» 

knp  in  Timerx  does  iKA  attempt  to  lea^  GetNextTpadQ:  The  coum  down  knp  in  Timerx  attempts  to  tead^ 


AGREEMENT  PROCESSOR 

Agreement  Processor  -  Process  Dependencies  .  117 

AgreeProcessor  Specification  .  118 

Determination  of  Token  Originat(»‘'s  Failure  .  119 

Processing  Agree  Tokens  .  120 

Agree.c  .  121 
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AgreeProcessor  for  agreep/p^)  at  p, 

1  if  (not  blocked  by  CommitProcessor) 

2  if  (initiate  agreement  message  received)  /*  p,  =  p^  */ 

3  add  agreepj^Pi)  to  TokenPool(p) 

4  -^PiCPt)  <-  joinagreed  or  failagreed 

5  send  agreepfp^  to  cwnbr{^) 

6  send  acknowledgment  to  calling  process 

7  else  /*  a  token  or  external  token  pool  is  received  */ 

8  if  {ExtTokenPoot) 

9  for  Vtokens  e  ExtTokenPool 

10  if  (token  e  TokenPool{^)) 

11  if  (originator  failed) 

12  ProcessToken 

13  end 

14  else/*  token  not  in  TokenPool  */ 

15  if  (received  for  the  first  time) 

16  ProcessToken 

17  end 

18  end 

19  end 

20  dse  /*  a  token  was  received  */ 

21  if  (received  for  the  first  time) 

22  ProcessToken 

23  end 

24  end 

25  end 

26  end 

Figure  A7  Agreement  Processor 
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LostAgreeToken 

1  if  (joinagree) 

2  if  (rankiPj)  >  rank(p)) 

3  return  true 

4  else 

5  return  false 

6  end 

7  end 

8  if  (failagree) 

9  if  (RetativeRankip^ ,  p,)  >  RelativeRank{p. ,  p^)) 

10  return  true 

11  else 

12  return  false 

13  end 

14  end 

Figure  A8  Determination  of  Token  Originator's  Failure 
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ProcessToken 

1  if  (joinreqst) 

2  send  token  to  JoinProcessor 

3  dseif  (commit) 

4  send  token  to  ComUProcessor 

5  elseif  (agree) 

6  if  ((p-  *  Pj)  &&  (agree  token  €  TokenPool(p)) 

I  add  agreepy(p*)  to  TokenPool(p) 

8  -STViCp*)  <-  FailAgreed  or  JoinAgreed 

9  send  agreCffp^  to  cwnbr(p) 

10  else  p^ 

I I  if  ((p.  =  p;  II  (Vp,  I  p,-^p.,  p,  6  sr„)) 

12  compute  rank  Vp,€  with  Agreed  status 

1 3  if  rank(pt)  =  smallest 

14  send  initiate_comit  to  ComitProcessor 

15  else 

16  ■S7'in(Pt)  <-  joinpendg  or  failpendg 

17  end 

18  end 

19  end 

20  end 

Figure  A9  Processing  Agree  Tokens 
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*  Get  socket  descrqjtcrs  from  command  line  aigumenL  myaddr  is  (piddr.&onubaek) 


=  in_msg_type(buO; 


)  /*endPh»essT( 


if  ^  wniemsg(coinfd,  (kninsg,  msglen)  <  msglen)  { 
printfCNlPiQoessToken:  llai  send  to  oommii  process  iailedNi"); 


if  ( writemsgOpfd,  tkiunsg,  msglen)  <  msglen)  {  ^**************i 

priiufCNnocessToken:  tkn  send  to  join  pnxess  faiicd>«");  ExeculeAgieemeni: 


It:  parsing  failed  for  td(en  subjecLNn"); 


tknl,&ong,3)— 0)  {  /*  Tin  cwiginalor  */ 

cuteAgreement  paning  &iled  for  token  originator.^");  if  (s&anp(orig,  myaddr) = 0)  { 

initoonunitphase=TRUE; 


nextiank-H-; 


6ee(member);  if  ( get&oinlist(lplknl,  &tplknsiibj,  2) = 0)  { 

printfCNExeculeAgieanenu  usken  pcxd  parsing  for  subj  of  token  %d  failed>n",  i); 


if  (strcmp(sialus,  "failagiee")  =  0)  /*  type  */ 


separator  */  printf('NExeculeAgreement:  inccnect  status  l%sf«i",  status); 


else  (/*  IstnaOest ->  update  status  with  appropriate  pending  status  */ 


if  ^  £rtfirmKa(ifJ{f«li!ia ,  &iokentype,  2) = 0 )  { 

(kies  the  token  pool  and  the  status  table  fcr  token  after  adding  pdntfOtlntegrateToken:  kxsd  token  oipy  parsing  for  lokeniypefailed>n"); 


I  /*endlntegrateToken*/ 


COMMIT  PROCESSOR 

Commit  Processor  -  Process  Dependencies  . 132 

Actions  for  Committing  a  Change  . 133 

Actions  for  Committing  a  Change  .  133 

CommitProcessor .  134 
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CommitChange  for  commtrp^(pt)  at  p. 

/*  Depending  on  whether  a  join  or  departure  */ 

1  add  or  delete  p*  from  CV(p.) 

2  delete  p^  entry  from 

3  )  «—  vn(p;)  +  1 

4  delete  all  commit  tokens  received  before  agreCp^p^)  from  TokenPoolip) 

5  if  (join  committed  &&  Joinre^fpjip^)  e  TokenPool(p)  ) 

6  delete  joinreqpjip,) 

7  end 

8  add  commitpfp^)  to  TokenPoolip) 

9  delete  agreCpf^p,) 

10  if  (current  host  =  p*) 

1 1  determine  new  p*^,, 

12  end 

13  if  ((join  committed)  &&  (p^,  =  p,)) 

14  send  STp^,  TokenPooKp),  and  GV(p)  to  acwnbr(p) 

15  end 

16  send  commitpjiPi)  token  to  cwnbr(p) 
end  CommitChange 

Figure  All  Actions  for  Committing  a  Change 
ProcessCommitTkn  for  commitpfjp^  at  p. 

1  if  (initiate  commit  message  received) 

2  generate  commit  token 

3  token  to  be  processed  <-  generated  token 

4  else  if  ((p.  ^  pP  &&  (not  duplicate)) 

5  token  to  be  processed  ^  received  token 

6  else 

7  exit 

8  end 

9  CommitChange 

10  while  (  p,  e  5r„  with  pending  status  &  Rank(p,)  <  Rank(p  J,  p„  e  STp;) 

1 1  generate  commit  token 

1 2  token  to  be  processed  <-  generated  token 

1 3  CommitChange  in  rank  order 

14  end 


Figure  A12  Generate  /  Receive  and  Process  a  Commit  Token 
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Get  socket  file  descriptors  fiom  the  cxxnmand  line,  myaddr  is  (ipaddnirant;back) 


I  local  token  pool*/  /*stripofrihe"tokentolai"  field*/ 

= GefTokenl^nKipmsoc);  imp.loken  =  CALLOC(  stikn(buO  + 1 .  char); 


ling  token  */ 


if  (Ftacess  taken)  {  f*m***m*<¥nt*m<ti**m*******m***m******^i********m***i******************tiiaiutct 

onkr.pending:  cieatas  a  linked  list  of  the  commit  {xooesses. 

head  =  onler_pendHig(lpooi,  gv,  soabie,  taken,  myaddr); 

while  Otead  !=  NULL){  node  ^)tder_pendiiig(tpool,  gv,  sttable,  kdcen,  myaddr) 


fieefsttableX  /*  get  oonunit  taken  subj  *! 

if  ( gelhomlist(d(n_list,  &subject,  2 )  !=  2 )  { 

)/•  while  (TRUE)*/  printii['\npcx£!ommitPBnding:  taken  subjea  parsing  failecNt"); 


status  table  -  add  all  members  to  list  in  raidi  order  V 


) /*  end  if  suhis  !=  joinfqstd  */ 


«]ti]t>*****««*>l<«4i4i4i4»«>)i**>*<***4i>t<*********«*>l‘**itc*i|c«4<*4i**«!t<:ti«4c««***i<i*4c«iii******«*« 

void  Comn[uLChange(lpool,  loken,  myaddr,  gvmsoc,  tpmsoc,  sunsoc,  intmbisoc,  foontsoc) 


if  (  gctfiomlist(ikiist,  &origij]alor,  3 )  !=  3 )  ( 

retuinfcmp);  prinlf("Coinniit_Change:  token  originator  parsing  failedSn"); 

printf(”W$$$$$$$$$m$$$$$$$$$$$$$S$$S$$$$$$$$$Sfl 
1  /*  end  makc_nodc  */  exil(- 1); ) 


if((sm3np(lktyperfailoonur>==0))  {  /*  assemble  the  f^joinicqst  token*/ 

/*  assent  ii|)daie  view  (deiele)  message*/  faux_loken  =  CALLOC(TOKENLEN+ l.char); 


ckMe<gvmfd);  if  ( GelMembWithRank(gview,  &host,  0)  !=  0 ) , 

printff  \Commit_Qiange:  gei  host  faikxNi" 

/•  ^  updaed  gioup  view  */  piintf(”W7$$$$$$$$$$$$$$$S$$$$$$$$$$J$l  $$$$$$$$$$«"); 

gview=GetGfDijpView(gvmsoc);  exit(-l); ) 


if(wniBinsg(inimlxfd,sndin4^msgien)<ms^  { 
pdntfCNCoiraniijChange:  sndiitipar  tx  failecNi"); 


prinlf('\ProcCtommiiPending;  token  subject  parsing  failetNi"); 


lplist,&iixakn,i)!=i)  { 

<3omiiiilRax&ng;  token  pool  poising  failed  for  token  %d^n",i);  sncal(deltkn,  nxlorig); 

3lsize=%(Ni",lpsize);  sticat(deltkn, 


litl^nding;  next  token  origiiiatflr  parsing  failecNi"); 


INTEGRATE  MEMBER  PROCESS 


Process  Dependencies  .  145 

Process  Specification  .  145 

Integrate  Member  Code .  146 


144 


Figure  A 13  Integrate  Member  -  Process  Dependencies 


IntegrateMember 

1  if  (initial  parameters) 

2  send  blocking  message  to  status  reporter 

3  send  GV  to  group  view  manager 

4  send  unblocking  message  to  status  reporter 

5  send  ST  to  status  table  manager 

6  send  TokenPool  to  token  pool  manager 

7  else 

8  get  GVp^  from  group  view  manager 

9  get  STp-  from  status  table  manger 

10  get  TokenPoolip^)  from  token  pool  manager 

1 1  assemble  initparam  message 

12  send  message  to  new  member 

13  end 


Figure  A14  Integrate  Member  Process  Specification 
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Mente  (INTMBR) 


/*  convert  token  pod  to  list  as  only  member  V 


invalid  fonnat  (token  pool]^n"); 


sm:py(msg.”siaiiGqs^');  gvm^  =  sn21isl(gvmsg,’Ni#"); 

msglen  =  sttlen(ni^;  stmsgl = sli2lisl(smisg,' W); 


JOIN  PROCESSOR 

Process  Specification  .  152 

Join  Processor  Code  .  153 
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Initiatejoin  for  a  join  request  message/token  for  at  p, 


1  while  (true) 

2  if(p«.«  57’„,GVp,) 

3  receive  join  request  message  or  token  for  p^ 

4  end 

5  if(P.  =  P*<,J 

6  send  initiate  agreement  message  to  AgreeProcessor  for  p^ 

7  block  until  AgreeProcessor  acknowledges  end  of  processing 

8  else 

9  •5^Pi(P«w)  ^  JoinRequested 

10  if  Ooin  request  message)  /*  p^  locates  p^  and  sends  its  join  request  */ 

1 1  generate  joinreq^f^^  token 

12  end 

1 3  add  joinreqffjp^  to  TokenPooK^) 

14  send  joinreq  token  to  cwnbt{p) 

15  end 

16  end 

end  Initiatejoin 


Figure  A15  Processing  of  a  Join  Request  Message  /  Token 
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c  =  argvl91;  lai:^lmembeitj]  =  NULL; 

iame=argv(101; 

=  argv(lll;)  /*  assemble  a  request*/ 

sirq^Ooinrequest,  "joinreqsfsi"); 


exil(-l);)  /*  no  changes  required  lo  the  initparammsgicvd  Grom  fionl  earlier'*'/ 


)  /*  end  while  foundmember  is  &ise  &  CountDown*/  )  end  else  foundmember'V 

}  r*  end  while  foundmetnber  is  &  groupsize '*/ 

)/*  end  else  CopyG  VFile  */  removelisl(sitelist); 

)  I*  end  while  foundmember  is  &lse  &  i++  */ 


=  sli2iist(lmp_buf,’Ni#");  stiq)y(token,  buO:  ) 


'( ge(fian]isl(myaddrlist,  Amyname,  l)!=l)l 
prinif('>tCopyGVFile:  my  ad±  parang  failedNn"); 


if  ( sBcinp(inynaine,  ip_address)  !=  0)  { /*  need  to  auempl  copying  */  ^*********************»i*********************************«um<** 

sprintfi[coniinand.  "nh  %s  cat  %s  >  %s" ,  sitename,  gioupname,  gioupname);  GetCountForJoin:  Detennines  the  wait  befoie  the  join  request  is  repeated. 


)  /•  endCbpyGVFife*/ 


DATABASE  MANAGERS 


Group  View  Manager  Process  Dependencies  .  160 

Status  Table  Manager  Process  Dependencies  .  160 

Token  Pool  Manager  Process  Dependencies  .  161 

Group  View  Manager .  162 

Status  Table  Manager .  169 

Token  Pool  Manager  .  174 
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COMMIT 

View  Request  |  PROCESSOR 


L-G!oapyj,w_  .Grou^yicw 

^tittLGroupView^  MANAGER  1^  View  reaue 


i 


INTEGRATE  ^  Group  View 
MEMBER  ' 


Group  View  J  MONITOR 
PROCESS 


Figure  A16  Group  View  Manager  -  Process  Dependencies 


Figure  A18  Token  Pool  Manager  -  Process  Dependencies 
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Qroup  View  Msnager  (GVM) 


/*  initialize  the  ^oup  view  for  a  single  member  *'/ 

inii_element = str2Iist(tmp,"^"); 

initialize(&head[,  iniLdement,  &view_nmbr,  &view_size); 


‘.(azis  MdiA'jqitiu  msia 


(0 = („I3P..‘is3nbai)duians )  ji 


|uest :  assembte  the  list  into  a  suing  aid  write  the  message  to  if  ( (writansg(soc^%/nsglen))  !=  msglen)  ( 

K  socket  pnnifC'G  VM  process.request  error  writetns^"); 


sncal(m5g,"r);  }  /*  end  else  head  */ 


I  /*  end  while  found  */ 


(getficnilisl(insgl,  &daia,  2))  =  0)  ( 
inlf("GVM  initiali/e  encr  invalid  format  cV); 


STATUS  TABLE  MANAGER  (STM) 


/*  Read  message  */ 

if  ( (msglen  =  icadmsg(newsoc,  &bur,  "#"))  <  0 )  ( 
prinlfC'STM:  read  enoiNn"); 

exit(-l); ) 


r  check  for  delete  status  V 

)  rend  while*/  if{strcmp(ncw_status"delstalus")=0) 


•«*«<ii*»**»*4i4i*«*«>**4>«*«**M«*>****4i»4i***«««>t»K««4i4>»«*»**  if  ^  (\vTitansg(soc  msg.  msglcn))  !=  msglen)  ( 

the  list  inio  a  string  and  write  the  message  10  printfC'STMprocess_iequestem]r  wiilcmsg"); 


)  /*endadd*/ 


»»»»***»<i«*»<i»»»**»»»*»»»»*<ii»*«*»**»»*»***»»****»*=»»*»****»»»»»*»*»»  if  ((next  element  -  2)  <  *si2e)  { 

iniriafaf!  ihe  gioup  view.  Delete  and  removes  existing  list  piintfC'STM  initialize  enor  invalid  fonnafn"); 


[m^  Adaia,  next_element))  >  0)  { 


POO- MANAGER  (TPM) 


printf(”'07; 
cxil(-l); ) 


(  ‘-(I')!"® 


i))!=nisglen)j 
enor  wriiemsg"); 


« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

* 

« 

* 

« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

« 

# 

« 

« 

« 

* 

« 


m  1  If  ^ 

S  ,s  I  s,  «.  I  i- 


'i 


)  /*  end  while  found  &  pir  */ 


fm  ^  ||^  OffSCl  ilUO  the  token  pocd  list  */ 

nitialize  the  table  view.  OeleleandieiiK>vesexistuiglisL  nexi_elefnent=2; 


DATA  CRUNCHING  PROGRAM 
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’(1 28.  char); 


lest_fd  =  fqKii(filename,  V);  et  =  sec  *  lOOOOOO + usee; 

if(lesl_fd  =  NULL){ 

printfrCRUNCR'  aoempt  to  open  filename); 


r>f9gieeinit->skipaline  V  /*  total#  of  decreasing  changes*/ 

if  (slian|)(type2.  "inil")  =  0)(  for  (i  =  1;  i  <=  niax_mbr  - 1;  i++)| 

if  (&canf(lBSLfd,  "%s%i%i%s%s%s",  type,  &ser,  &iisec,  type2,  subj,  origin) = BOF){ 
printfTQlUNCH:  reading  lest  file  at  host);  siim=0; 


t 

>< 


is 

E. 


II 

>< 


i 


/*  parse  the  extra  data  for  the  agree  recv  */ 
while  (fgelc(test_fd)  !=  I0){ 


the  coi™!  send*/  /*  close  the  lest  file*/ 

if  (&canf(tBst_fd.  "%s%i%i%s",  ^pe,  &sec,  &iisec,  ^pe2)  =  EOF){  if  ( fclo86(lest_f(l) = EC»v  | 

priitfrCRUNCR-  leaduig  lest  file  at  host);  printfC’CRUNCH;  attempt  to  close  lest  fileV); 


=  sec  *  1000000 + usee  - 
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